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ABSTRACT 


The problem is Hliefani@d as examining a list of the 
research brojecdaprbluently sponsored by the Department 
of the Navy under GOR-43 and analyze the ordered project 
listings as ranked by panels composed of persons with 
interest in those projects. 

Data consisted of project priority rankings gf IG 
judges composing 9 separate panels, each judge ranking 
the projects twice in priority order; once considering 
the practical significance of the project and once con 
sidering the possibility of a successful research effort. 
Project rankings of judges within each panel were scaled 
using the FORD procedure and the 9 panel rankings then 
investigated for similarities using methods of cluster 
analysis. 

The results showed that no 2 judge panels submitted 
highly correlawed rankings of the 18 projects, but that 
under each criterion several project clusters were evi-. 
dent, indicating equal research emphasis desired by the 


panels. 
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Teen RODY GL LON 


There exists within the Department of the Navy, many 
organizations with varying convictions about which Navy 
Personnel Research and Navy Training Research Programs 
should receive priority when allocating research funds 
and research time. The diverse opinions of program empha- 
sis are due partly to the different oC DOLE OE Of thie 
Pproups sy partly to the dafftierent functions! of the»groups 
and partly to the relationships of the groups to other 
organizations both inside and outside the Department of 
the Navy. 

Undoubtedly, those who make the final decisions have 
the greater number of relevant facts upon which to base 
their decision. They probably exert greater effort to 
Bic om odie rocess itself than their critics 
might. Thus their decisions should be the best "all fac- 
tors considered”, decisions. Nevertheless, an examination 
Peer cmorvemsi ty OL Opinions among different groups 
Should help their decision-making process by providing 
additional information and by pointing out areas where 
closer agreement among various groups would promote the 
research effort. 

The objective of this paper is to draw, from various 
groups, data concerning which programs they feel should 


be studied, analyze the group decisions and examine 





Similarities and dissimilarities of their priority pro- 
ject choices. 

The nature of the problem is two-fold. 

1. Compare the groups by comparing their choices of 
which projects should receive priority when allocating 
research laboratory resources. Establish communities of 
judge groups whose priority rankings displayed a degree 
Om escamirlarity. 

fee EXaMi nema wldst of projects cunrently being 
studied by the Bureau of Naval Personnel and associated 
research laboratories. Determine if there exists projects 
which inter~-group consensus shows should receive either 
moe Or low priority when allocating research laboratory 
resources. 

The method chosen to compare the judge groups was by 
means of a two-dimensional plot where inter-point dis- 


MemecsS ONsthe eplotsegre almonotonic function of the judge 


paeOQup Similarities. More specifically, 
> => < 
ty ik “ig ~ Sak 
ag = a correlation measure between groups i and j. 
cog = the inter-point distance for points i and j on the 


two-dimensional plot. 
By examining the resulting plots using methods of 
cluster analysis, those judge groups whose project prior- 


ity rankings compared favorably were readily evident. 





mhe®methods for comparing the various projiectis were 
the same as for the judge groups except that a dissimi- 
larity vice correlation measure was utilized for inter- 
ebyect comparison. The technique for determining; the 
plot accepted either statistic as input. The resulting 
plot displayed the projects as an inverse monotonic func- 


tion of the’ dis’simillarity measure, Speciffically, 


< => < 
eg tj ik 
Sat = the measure of dissimilarity between projects i and 
j. d,., = the inter-point distance for points i and j on 


1j 


the two-dimensional plot. 

Methods of cluster analysis discerned those projects 
Which received similar rankings on the priority scales 
Oerethe judge groups. 

Comparison of the mean ranking for each project over 
mmmepudrzce sroups showed which projects were ranked high, 
Mowe, etc. The variance of the mean ranking for each pro- 
ject indicated the consistency of its ranking by the 
judge groups, e.g., the projects with the lowest rank 


variance were ranked most consistently by all groups. 





LL. DATA DESCRIPTION 


The data source for this study consisted of approxi- 
mately two-hundred individual judges, each ranking the 
same eighteen projects on a priority ladder. That pro- 
ject which the judge felt should receive top priority was 
ranked first; the second most important was ranked second, 
etc. In addition, each judge was allowed to indicate 
programs which he believed should receive equal emphasis 
by placing them (up to six) in the same group. To in- 
crease the validity of the data, judges were asked to 
rank only the projects which they felt knowledgeable 
about and competent to rank. 

Each judge produced two priority ladders, one ranked 
Using the criterion of Military Significance of the pro- 
meeimeandeone Tanked using the criterion of Investment 
Risk of the project. The two criteria are explained in 


the following paragraph. 


A. PROJECT RANKING CRITERIA 

The two major considerations when deciding upon which 
projects to expend research resources are the potential 
Military Significance of the program and the Investment 
Risk of the program. The judging groups of this study 
were asked to construct priority ladders of the con- 


Sidered programs under each of those two considerations. 





iy, Certerion A, Military Stenificance 


Military Significance refers to the needs of the 
Navy. The projects were ranked on the basis of which 
would yield the greatest payoff to the Navy if the re- 
search was successful. Those projects thought to have 
high military signifiicance were ranked high and So on, 
down to those of low military significance. The proba- 
bility 9f a solution being achieved was not considered 
under this criterion, only the significance of the solu- 
tion if it were achieved. 

Ze Critenion Baw. Investment.Risk 

Investment Risk refers to the probability of a 
Successful research effort regardless of the value to 
mm@emavy of the payoff. The project orderings repre- 
sented each judge's belief of which problems had a high 
probability of being successfully solved (ranked high) 
me «Which had a lower probability of being solved (ranked 
lower). A research effort was defined as successful if 
a solution to problems or significant improvements in 
the present state could be developed within a period of 


Bem years. 


B. THE JUDGE GROUPS 

The judge groups chosen for this study represent 
those organizations which are vitally interested in the 
allocation of research resources to the various projects. 


The list of the nine groups is contained in Appendix A. 
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They represent a cross-section of persons directly con- 
nected with the Bureau of Naval Personnel and with the 
Naval Personnel Research Laboratories. Both managers and 
professional workers are included as judges. In addition, 
group number 7 is composed of persons who are users of 
research products; group number 8, evaluators of research 
erfortis; amd group 9, contwmacted workers om resejiarch pro- 
grams. Accordingly, none of these last three groups is 
directly involved in selecting projects for emphasis in 


research. 


Cc. THE PROJECTS STUDIED 

The projects selected for study are ones of current 
Digwemn pe rest swtthin thte*® Department of the Navy. Appen- 
di-weretrsctSoMmana descripes the projects. One of the 
objectives of this study was to determine if the judge 
oman emietLiateptogram priority lists should be re- 
assessed. Thiet projects considered represent all those 
that Personnel Logistics (General Operating Requirement 
43) is currently concerned with, plus the additional 


project of R & D Resources Management (No. 16). 


Alea 





IIL. CALCULATIONS ON DATA 


The raw data submitted by each individual judge was 
in the form of two priority ladders, one for each rank- 
ine criterion. To vfacilitate comparing the judges groups, 
a Single composite ranking of projects for each group, 
which represented the preferences of each judge in that 


group, had to be derived. 


A. THE FORD PROCEDURE 

The Ce aenriaae for developing a composite Pant n c E Ons 
each judge group is known as the FORD PROCEDURE [Ref. 1]. 
Basically, the program determines a weighted value for 
each project. The weight statistic is calculated from 


a win-loss matrix by solving iteratively the equation 


(Ol _ [ie 
s ce. te HER. 
a a 
J we te Ww 
a. J 
where 
ae = the number of times project i was preferred to 


Project }) by all judges in the group. 
we = the@weighted value alsctpnei  towprojcereteon, Chesil a 
gteration. 

1 


of the win percentage Eor preojectmiwon the first itera— 


E11 Onl. 


2 





It should be noted that the FORD PROCEDURE produces 
a composite judgment that reflects the contribution of 
individual judges according to the number of judgments 
he makes. 

Several commendable characteristics which illustrate 
why the FORD PROCEDURE is especially applicable to this 
Study are, 

1. A judge or ranker needed not rank all the pro- 
jects, only those he felt knowledgeable about and compe- 
tent to rank. 

2. Each judge could make his ranking as coarse or 
as fine as he desires. There was no restriction on the 
number of judgmental categories he may use. 

3. <A judge had no requirement for a fixed distribu- 
tion of projects among the categories except that no more 
bmemeoime-LM@ird Of the projects could be placed in any one 
eatermory. 

feemgudge could indicate, projects of tied ranking 
by placing them in the Same category. 

The input to the FORD PROCEDURE, as run on the IBM/ 
360 computer, was composed of each individual judge's 
ranking of some or all of the eighteen projects, ordered 
under Criterion A or B. The program Scueiderfd all judge 
rankings of judges in one group and calculated a weighted 
value for each project. By ordering the projects with 
the highest weight first, a single composite ranking of 
[eee tS representing all the judges in that group was 
obtained. 
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B. DATA MATRIX TRANSFORMATION 

After computing a composite ranking for each group, 
the data for analysis had been reduced to an 18 x 18 
matrix M. "Rank Deiter: | were labels on the ordinate 
and ‘judge groups' were the labels on the abscissa. One 
additional ranking was added to the data matrix as column 
uO". It represented the projects ranked high to low as 
reflected by current fiscal year funding. Cote 1 
through 9 of matrix M represented the priority rankings 
OfPthe® pro jereat se by Ehfes judge groups under the A criterion. 
Volunne ws lletiimauUcnmLoerepresented the priority rankings of 
the judge groups 1 through 9 respectively, under criterion 
B. Element Me showed which project was ranked in the 
i th position by judge group j. Appendix C contains 
matrix M. 

fomenapwenwciastee Calculation of correlation coeffi- 
cients between judge groups and between projects, the 
matrix M was transformed to a matrix N, 18 x 19. Matrix 
M had ‘project number' on the ordinate and ‘judge group'- 
on the absicssa. Thus the element aE, represented the 


ranking of project i by judge group j. 
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C. CORRELATION BETWEEN JUDGE GROUPS 
From niatrix N the correlation between judge groups 


was calculated using the formula 


2 

. 7. OG) 

ij k J 

Z 
N(N =-1) 

where 

od = the difference between the ranking of project k by 
judge groups i and j. 


the’ number of ranks. 


N 


k amendex to tiacililaitate stéEpping through each of the 


18 projects. The correlation coefficient ae has the ad- 
Manure of not requiring any assumption on the distribu- 
mlomeot the data. Matrix R in Appendix D displays the 


r.. values. 
1J 


D. PROJECT COLLATION 

MESiLaitiigtasamwas required that would quantify the rela- 
tionship of Bro ieee mero projy;ect |). It was decided that 
ime’ best method was to compare their respective rankings 
by the judge groups. By calculating a difference in the 
mean ranking of each project, no consideration would be 
given to the variance of the rankings. Note that if pro- 
jects r and s with mean rank difference equal to X have 
small rank variance, they are ‘farther apart' than projects 


u and v with a difference in mean ranks equal to X and with 
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large®rank varianae. Therefore, the jstatistic t was 


ij 

developed and used as a dissimilarity measure. 

Mie eel. 

a 
“ Cay = 
Veuve cu, 
(STD, ) (STD), 

where 
MR, = the mean rank of project i over the judge group 
rankings. 
STD, = the standard deviation of the rankings of project 
i. Project mean ranks, standard deviation of the rank- 


ings, and the tabled ig statistic are shown in Appendix 
E. The smallest value for standard deviation of project 
ranking was 1.68; therefore, the problem of a very small 
—~STD, value disproportionately affecting the t,. Se ais — 


Game was’ not encountered. 
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Tv MULT TDIMENSTONAL SCALING SANALY SS 


The method chosen to compare the various judge groups 
and to compare the projects was multidimensional scaling 
analysis. The particular technique employed was non- 
metric Smallest Space Analysis (SSA), and has evolved 
through studies by Shepard, Kruskal, Guttman and Lingoes 
[Refs. 2-6]. The computer program used 58 developed at 
the University of Michigan by Guttman and Lingoes and is 
designated Smallest Space Analysis - I (SSA-I) [Ref. 7]. 
ious Linaseamset Of coordinates for each of the n 
points (n Fudge groups) in an m dimensional Euclidean 
Space such that the distance between points is a mono- 
Pommemrunection of inter—judge group correlation. More 
precisely, given a set of eae ic te Pomewn iene th Cace a Sea 
G@emeomed disitance function, such as object correlation 
ranks he omeeticmmin—1)/2 pairs of pointsi, find that set 
of real numbers, Xan (eel, se, Nepayp= 1, 2,.., mm 


such that if 


then at 3 < Gl The concept of the 


whenever Pas < apo 


Feil 


distant function oie was important to the scaling analysis 


and deserves amplification. When reading as computer 





Lingoes, J. C., New Computer Developments in Pattern 
Analysis and Nonmetric Techniques, paper presented at 
the 1964 IBM Symposium of Statistics, Paris: Gauthier- 
Villas, 24 November 1966. 


ey, 





input the correlation matrix R, the distance function Oe 


refers to the rank of correlation coefficient ase in an 


ordered column vector V. For example, the highest ro. 


J 
value was assigned Pay = 1, the second highest ssi was 
‘ a . ° 1 < 
assigned ig 2 etc The meaning of ane dia whenever 


 Siise them obvious. 


Pig ~ PRA 
Thempwacticesot utilizing only the®*rank’s of thle co- 
efficients and not the actual values when determining 
optimal point orientation on the multidimensional plot is 
the nonmetric method. Nonmetric rather than metric con- 
straints permit bypassing a cork linear and distributional 
assumptions required of metric methods. Defining a true 
Metric on psychological dee as was the case in this 
problem, was impossible. To enable the plotting of a 
meaningful display the number of Euclidean dimensions is 
mmited to m = 2. A true metric constraint denies that 
possibility due to the strict linear distance measure- 
ments between points. A nonmetric monotonic distance 
function allows for the reduction to a two dimensional 
display while still maintaining a meaningful plot. 
Several goodness-of-fit coefficients used in this study 


confirm the nonmetric assumption as acceptable. 


A. SMALLEST SPACE ANALYSIS (SSA-1) ALGORITHM 
The algorithm for determing the optimal orientation 


of points representing the n objects proceeds as follows. 
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STEP J: kKead in thie data matrix R of inter-group 
correlation values or the data matrix T of inter-project 
dissimilarity values. 

STEP 2: Sort the data values from large to small if 
similarities and small to large if dissimilarities. 
Assign ordered ranks of 1 to n(n-1)/2 to the ordered 


vector. Indicate the rank of element eae by ae 


pHP 335 Comstruct a matrix C, by 


i p,,/fn(n-1)/2] al 
(3 = = n 
ij . 
n >» C1 i= j 
saa 
Le 
pOlve for the Eigenvectors and Eigenvalues of C. The 
set of Eigenvectors Xan Cie eee... Gna EP 1, 2,45) 


moa the initial orientation of the n points in the m 
mamensionali Euclidean space. 
STEP “Gee Catlculate the inter-point distances over 


the m dimensional space. 


m 
a 231 2 


Order the distances d,, in a row vector with n(n-1)/2 
elements. 

STEP 5; Redefine’ the’ o%s? ads! the’ d “si permuted to 
maintain the rank order of the original distance function 


and designate them d*, 
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STEP 6: Calculate a measure of effectiveness, the 


pha coefficient. 


n-l n 
* 2 
sa Op De Cd) 
i=l j=1+1 J J 


SLEE 7s GCavlculalgel a?) new Co matrix on the balsifs off 


the, d'Swamd theyd*"s. 





n 
se 
1 + Sy d ./4 k i=j 
k=l 
= — es 
“ij d.. 
= — eS 

hf ck 

se 

x dig 

mmare d.., = 0, d., = 0, and amo by “convection. 
‘al anit ef 
ii 
STEP 8: Calculate a new set of point coordinates 
n 
xis = l/n Dy eee = 1 
aie te ’ ik “ij 
lige 
DBIEP 9 Calculate again the inter-point distances 
memeper Sittepm¢ and repeat stieps»s5, 6, 7 and 8. Continwe 


until the normalized phi coefficient indicates further 
adjustment of points will not advance the state of the 
inter-point distance and original distance function 


Bellatlonship toward actual monotonicity. 


B. GOODNESS-OF-FIT MEASUREMENTS 


i Guttman-Lingoes Coefficient of Alienation 


Consider a scattergram of n(n-1)/2 points on 


tt. 


nw 
coordinates oa e Vs See ao Maximitmine agstat istic U7 
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is defined as minimizing the alienation of plotted points 


from a regression line through the origin. The coeffi- 


etent, of “alienalttion is*Kke= (1 = Wye, The sitaea'stic 
* 
is a rank-order correlation coefficient between Ps = tes 
de 
and d.. * uw equals 1 when d.., = ad... for some constant 
ij iJ 1] 


Gt iproportionality aiand it WPollows thiat thle coefficijent 
of alienation K would equal zero. 
2. Kruskal "'s' Stre'sis 

Kruskal's stress is derived from a two-dimension- 
al plot of diy Vs Oss and is a statistic quantifying how 
well the given configuration of points represents the 
data. A monotonic increasing piece-wise linear line is 
duawmeconneeting the points. The points di, are also 
plotted on the graph. The distance bese - di) is defined 
as the horizontal difference between the actual distance 


* 
o Ae emaethe desired disitance te . Kruskmal's stress ils 


defined as, 





* 2 
Pai) (a.. - a.) 
S = wee OT) 


2 
me (4y,) 


Application of the statistic has shown that for any value 
Oreoetesse than .20, the plotted configuration of points 
is a good representation of the original data. Zero 
stress indicates complete compliance with the monotonic 


Goustraint.- 
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Van, CALCULATION RESULTS 


Torwaccomnplish the? obtjiectives of Ehis? Study of "conpar— 
ing the gudge croups under vJeaich of they yanking criteria, 
and of comparing the projects being ranked, the Smallest 


Space Analysis - I was employed several times. 


A. JUDGE GROUP COMPARISON 
1. Comparison of Judge Groups under Criterion A 

Columns, 1 throweh 10 of mat ries tein whicn oe rep- 
resented the ranking of project i by judge group j under 
Criterion A (Military Significance) and column 10 repre- 
senting fiscal yéar funding, were taken for the? filasit 
calculation. The correlation matrix R was calculated for 
the measure of similarity between judge groups. et 
2,020,105 j=l, 2,..., WO as Siiemeusied was input to SSA-1. 

Table 1 “shiotwis® fivem weissaiiim: point postition’s §rep- 
resenting the optimal orientation of each judge group 
with respect to the other nine. Phepvdaiies for Guttman; 
Lingoes' coefficient of alienation of .290 and Kruskal's 
stress of .243 were both marginal values and indicated 
that an actual monotonic relationship between similari- 
ties and inter~point distances was not attainable. The 
plot of@gemnws, Hiomre boedid not indicate any clustering 


oo 


or significant similarity among judge group rankings. 
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TABLE I 


PLOT OF JUDGE GROUPS 


RANKING UNDER THE MILITARY SIGNIFICANCE CRITERION 


Judge Group Blot Go@oxd 7 mates 
iL Ol. 2 O16 99.664 
2 SUSI SS 6 each Is 
3 6.069 —-16.553 
4 -100.000 -3/.483 
5 SoUp2 oo On 79 
6 100,000 Se 2S 
7 19,227 -100.000 
8 Bon ulS 32, a0 
9 = 1d) 6 Ateye 20.02 3 
10 “99.613 -~7.008 
Guttman-Lingoes' Coefficient of Alienation = 0.29037 


Kruskal's Stress = 0.24326 
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Velctor 41 





Plot of Judge Groups Ranking under the 


Figure wae 


Militamsawenaiticance Critierion. 
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Za Comparison of Judge Groups under Criterion B 


* 


Columns LTOVthnrousche ly ofs matrix Wh with ae repre- 
Senting, the’ priority mank of project gi with Respect to 
fiscal year funding and judge groups 1 through 9 respec~ 
tively, under Criterion B were used for the second run. 
The correlation values Tay (i= 1] OSG eo: W10,11,.. . 9%) 
Erome matrix R of Appendix D was the*imnput to SSA-I. 

Ji bbowit Shows ther nesulting “point positions on 
the two-dimensional display. The values for Guttman- 
Bamcoes' coefficient of jialienation of .257 and for Krus- 
kal's stress of .214 were “ye mareinal. ihe plot of 
pemes in Figure’ 2, by being very scattered, indicated 
that none of the judge groups agreed to a significant 
degree on their project rankings. 

Ss. Compiirison of JjudgesGroups under Both Criteria 

Neither the plot of judge groups under Criterion 
mor thesplopewer judge groups under Criterion B indica- 
media strong similarity in pgjroup decisions. By plotting 
all 19 "groups" together it would be possible to see if 
any groups submitted very similar rankings under both 
eriteria. Correlation coefficients ts, (Se re ees Ge 
mo. 2,..el9) from matrix R of Appendix D were input to 
SSA-I. 

Table III lists the plot positions. The Guttman- 
Lingoes,' coefficient of alienatiom was) 7l90,and the Krus— 


Kal stresSm@wauselorva The Values mapeatedean excellent 


Ih S, 





TAB GEE 
PLOT CY JUDGESGROUES 


RANKING UNDER THE INVESTMENT RISK CRITERION 


Judge Group Plot Coordinates 
1 89.897 38.945 
2 33.497 gu on? 
3 = OCHO OOm—nrn= 4 5 4173 
4 ea) 2166 57.870 
5 33.039 69.747 
6 100.000 ~53.064 
7 a 16.167 -16.339 
g =23.906 -100.000 
9 man 639 23.583 
10 Siycoil = 112.3864 


Guttman-Lingoes' 
eer etent of Alienation = 0.25766 


Meamskal's Stress = 0.21421 
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TABLE III 


PLOT OF JUDGE GROUPS 


RANKING UNDER BOTH CRITERIA 


Judge Group Faloite Cocmd i majt as 

i ay 8G 24.269 

2 ~6.882 MOB 

| 3 51.030 ~-100.000 
4 “O10 ae ~24.549 

5 ea? 1. 317 =O 273 

6 ~8.888 RO RAY? 

7 -~64.799 250) 28 

8 Boge OG? -48.063 

9 ~100.000 294.551 

10 18.474 ~83.546 
11 74.592 ~7.040 
12 81.520 or O29 
“3 86.038 ~42.891 
14 59.298 -25.100 
15 66.162 -65.892 
16 - or, “A SOE 
17 ~9.576 ~16.639 
18 62.696 =71.514 
19 100.000 My Sse 


Guttman-Lingoes' 
Coefficient of Alienation = 0.19045 


Moe Ss Stress = 0.16754 
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DPlOumingi@eyuis of Variation from thesrerresspon lines for 
the G-L coefficient as detailed in paragraph IV C 2. The 
Kruskal stress also indicated excellent correlation of 
plotted values with the monotonically increasing line on 
Ps vs dy as explained in paragraph IV C 2. 

Figure 3 was not applicable to clustering consid- 
eration because it represented the judge groups as nine- 
teen separate entities, i.e., each judge group was 
represented twice. However, one thing was clearly indi- 
cated. Each judge group submitted significantly differ- 
ent rankings for each of the two criteria. With the 
Ssoaepelon Of group 3 under Criterion A (point 3) and 
group 7 under Griterion B (point 17), there was no over- 
maeoane of the judge group rankings. The results indica- 
moa that the judges actually did consider the criteria 
of Military Significance and Investment Risk when ranking 


mreir projects. That is an important argument for data 


Welidity. 


B. PROJECT COMPARISONS 
The Pecan objective of this study was to compare the 
projects and examine possible groups of projects that the 
judges considered of equal importance or that judge group 
consensus rated high or low. 
1. Comparison of Projects Ranked under Criterion A 
The eighteen projects as ranked under Criterion A 


plus the fiscal year @iundsier ankings wememthe Sirst of tine 
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f 


Peowucict wroupseconsidered. The Entries® of matrix Ty a7 
Appendix E are values of the interpoint dissimilarity 
measure Bae for the Military Significance crlieons 
Theb@input Go SSA-T was the matrix Tae 

Table II shows the point position representing 
the projects on the two-dimensional plot. The value of 
Guttman-Lingoes' coefficient of alienation was .067 and 
Kruskal's stress was .053. Both values indicate that the 
resulting plot was an excellent reflection of the origi- 
nal data as determined by thé distance function Aaa 
Figure 4 showed definite clusters of projects. The rule 
chosen ue defining a cuusten or c¢ points was that each 
DOtmeNDpCEWEEnan 20,0 untts®oft the®other c-l) points of the 
cluster. (The longest possible inter-point distance on 
Pio enorme ecogwUdanatsi,) Using that rule, 2 major 
clusters of 3 or more points were established. One 
ae ewasmcomposled Of projects 5, 8 and 18, and the 
other cluster was. composed of projects 3, 10, 12, 14 
and 15. Minor clusters were established between pro- 
jects 7 and 9, and between projects 4 and 16. The 
implication drawn from the clusters was that the pro- 
jects in each cluster were considered of equal priority 
by the joes groups when ranking under the Military 
Su@midltfoance Criterion. The SSA-I plot is more meaning~ 


Pia tereesiLanaardad correlation statistic because each 


point was plotted with respect to the other n-l points. 


Bal 


” 





TABLE IV 
PLOT OF PROJECTS 


WHEN RANKED UNDER THE MILITARY SIGNIFICANCE CRITERION 


Project No. Plot Coordinates 
di a0 .636 -46.802 
2 ~80.882 OG 
3 5.479 ~82.690 
4, me, 223 ~48.019 
2 ~34.760 ey 
6 -16.022 -75.868 
Z Se. i3 Oy. 7 O02 
8 ~41.815 =56.905 
9 SU 7 2 =) Og 4 Oo) 
10 om 92 woo. /19 
i T reo 0 Si SOHON- eP0) 6 
2 Oo 3 i le 
t3 100.000 ~26.187 
14 11.188 = 1 Ua Sens 
5 16.804 = J! -, Biable 
16 55.806 move 7 L 
7 ~100.000 ao lve? 90 
18 ~51.634 -49.,414 


Guttman-Lingoes' Coefficient 
of Alienation = 0.06728 


Kruskal's Stress = 0.05308 
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Figure 4, 
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A correlation statistic reflects the collation between 
two points regardless of each point’ s réelationghip to 
the remaining points. 
2. Comparison of Projects under Criterion B 

ThementrieS sole Matias 1x T. ineAppendix KE were juseid 
as the SSA-I input for comparing projects as they were 
ranked by the judges under She eee RasSkeGrdi ter lon. 

Table VY shows the point positions, the Guttman-— 
Lingoes' coefficient of alienation as .030 and the Kruskal 
stress as .021. Again the values indicated an excellent 
f2t withenespect to thre Oro Ginis constraints. Figure 5 
showed the project point orientation. Using the rule 
that inter-point distances be less than 20.0 units for 
each cluster point, three major clusters were established. 
One cluster was composed of projects 3, 15, 1/7 and Ss 
another was composed of projects 1, 13 and i. amd the 
third of projects 6, 7 and 10. A minor cluster between 
projects 12 and 16 was also in evidence. The reskore, rank= 
ing on the basis of Investment Risk the judges felt the 
clustered projects entailed similar risk. 

3. Comparison of Projects using Bowiee@rs ceria 

Inpaet ual practice both Military Significance and 
Investment Risk are considered when allocating research 
resources. For that reason a plot of the projects consid-= 
ering both criteria was: desired. Matrix TAR of Appendix E 
wermused as the input for SSA-lI. It was derived by con- 


sidering the judges as nineteen separate groups’. 
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fapbbe vi displays theppoint positions. The Gurt— 
man-Lingoes' coefficient of alienation of .026 and the 
Kruskal ‘streiss of .018 both indicated an excellent plot 
in terms of meeting the plotting algorithm constraint 
criteria. Figure 6 showed several clusters, the major 
ones beine wrojgectses, 8, 17 and@lejyeand projectis 4, 6 
and 7. Minor clusters conSisted of points 9 and 11, points 
Seands 155 points, 12 fand 145 ange pointis@leand 16. Consider- 
ings whew projects: umder both rankin! criteria, those 
clusters represented the projects which the judge groups 


felt should receive equal emphasis for research. 
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TABLE V 


PLOT SOF SPROIEC.LS 


WHEN RANKED UNDER THE INVESTMENT RISK CRITERION 


Proycet Now Plot Coordinates 
1 81.826 =e ss 
2 -85.361 “15, 9ue 
3 -~54.677 -~66.792 
4 -~68.344 -~52.507 
5 we -100)- 000 -~29,425 
5 18.818 -~74.158 
7  S8g ile -64.395 
8 -~67.295 a6 2 1.1 
9 -82.586 ~4.289 
10 =O) alae (7) oe 
ie 26.285 -~100.000 
12 | ral Get -43.641 
13 , 100.000 93. 82 
14 94.434 =D RE 
15 -48.304 =onle, 729 
16 | 68.200 -54.758 
17 ~45.503 =7O, 72a 
18 -43.436 -~64,363 


Guttman-Lingoes' 
Geerficient of Alienation = 0.03073 


Kruskal iq Stress, —2002190 
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Figure 5. Plot of Projects when Ranked under the 
Investment Risk Criterion. 
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TABLE VI 


PLOT OF BROJECTS 


WHEN RANKED UNDER BOTH CRITERIA 


Project No. Plot Coordinates 
1 84.667 -38.574 
Z -100.000 46.714 
é = 1133 atOre POL 
4 =U. J OW -83.865 
5 eg. / Oe] -35.684 
6 -37.145 =96. 417 
ay ae Ih, 8 (V8, =e} ly (PA, 
8 20.9 79 -66,.706 
9 5G 6 OO =-63.651 
10 213.578 200.000 
digil -66.832 -04 756 
a2 29.745 Sad Oy Site" 
aS 100.000 -9.750 
14 34.864 -67.964 
I moU. 375 ENO) 3 Sie) 
16 TIAA NS, =52.623 
ey SEN NS ET NS. 
13) mea Oo -59.449 


Guttman-Lingoes' 
Seeeracient of Alienation = 0.02633 


Mmeica)'’ s Stress = 0.01814 
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Plot of Projects when Ranked under Both 


Figure 6. 


memereria. 
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Vi. CONCLUSIONS 


Results of the SSA-I plots of judge groups indicated 
that no two judge groups were in strong agreement when 
considering their rankings as a whole. Groups often 
agreed on which projects should be ranked high under one 
criterion or the other but disagreed on which projects 
Should be ranked low. The reverse, agreeing on the lower 
ranked projects but disagreeing on the higher ranks, was 
also in evidence. 

Dit pememrmmeonciwvsions ean bei drawn from the project 
comparisons. The three plots using @ritdrion A, Crite-~ 
Pe oeancmoOlMeGrrtteria A and B, al’) displayed cluster- 
ing tendencies. To display the results in an easy-to-read 
fashion the following was done. For each cluster, the 
Hetmemaiken OL the projects in that cluster was calculated 
and designated the mean cluster rank. A ladder was con- 
structed of ordered mean ranks for both clusters and non- 
clustered projects. The result is how the judge groups 
felt the projects should be ranked considering the three 
criteria above. Figure 7 displays the resulting project 
priority ladders. | 

Investigation of the variation in project rankings 
(Appendix E) shows which project was ranked most consis- 
tent by the judge groups. That project with the lowest 


standard deviation of ranking was project 17 with a mean 
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Prank of 4.70 and &! standard devilaition of 1.68 for Crite- 
EHOMEAn project 13 With Aumiean rank’ of 1590 and a 
standard deviation of 1.8/7 for Criterion B, and project 
13 with a mean rank of 15.74 and a standard deviation 


Coe 7 tor Criteri as Amand Be Copmelthier . 
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Pro jeact mene te 

Education and Training 
Personnel Retention 
Recruitment/Procurement 


Manpower Requirements 
Selection/Classification 
Personnel Performance 


Personnel Management Systems 


Occupational Research and 
Development 

Catrieler” Structures 

Att mwtiudel Motivation 

Organizational Effectiveness 

Distribution/Assignment 


Man-Machine System R & D 
Applied Systems Development 


Manpower/Personnel Cost Systems 
Kee Keisources! Managiement 


Drug Abuse 


Minorlty= Discrimination 
Measurement and Reduction 


Figure /a. Project Priority Ladder as a Result of 
Clustering under the Military Significance Criterion. 
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Rank 


6 


Figure /b. 


Proj. No. 


bf oh bY fj 
Ary lona!l lowu wl o fF WU wo DN 


jt 
Lo ke 


14 


Prowdet Title 
Education and Training 
Applied Systems Development 
Manpower Requirements 
Manpower/Personnel Cost Systems 
Selection/Classification 


OCeupiac& LOnail Research and 
Development 
Distribution/Assignment 
Recruitment/Procurement 
Personnel Performance 


Personnel Management Systems 
Man~Machine Systems R & D 
Career Structures 


Penmaonnet Retention 


Attatwde/Motivialtion 
R & D Resources Management 


Drug Abuse 

MAinOGreye JI SCcCriminat lon 
Measurement and Reduction 

Oueenizational Effectiveness 


Project Priority Ladder as a Result of 
Clustering under the Investment Risk Criterion. 
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Rank Peo i Noe Projveecertitie 


I 2 Education and Training 
2 2 Manpower Requirements 
8 Selection/Classification 
7 Recruitment /Procurement 
18 Personnel Performance 


Applied Systems Development 
Personnel Retention 


Occupational Research and 
Development 
Distribution/Assignment 


Manpower/Personnel Cost Systems 
Personnel Management Systems 


Man-Machine Systems R & D 


Career Structures 


ee See 


7 12 Att teem die-/ Mot iviait ion 
14 Organizational Effectiveness 
8 1 Drug Abuse 
aie R & D Resources Management 
9 13 Mawes rt ys Discrimination 
Measurement and Reduction 
Paeeure 7c. Project Priority Ladder as a Result of 


femerering under both Criteria; Military Significance 
and Investment Risk. 
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Group Number 
1 


2 


10 


APPENDIX A 


LIST OF JUDGE GROUPS 


Group Composition 
Manager personnel, Pers A-3 


Manager personnel, Headquarters, Dept. 
of the Navy 


Manager personnel, Naval Personnel 
Pabormatoryyee washington, D. C. 


Professional worker, Naval Personnel 
Laboratory, San Diego 


Professional worker, Naval Personnel 
Poe oeratlony wewashington, Ds. C. 


Manager personnel, Naval Personnel 
Laboratory, San Diego 


Members, user panel (students at the 
Naval Postgraduate School) 


Members, Laboratory Advisory Board 
for Personnel Laboratories 


Technical/Professional persons, con- 
ELe@eror activities 


Piscal year 1973 dollar funding for 


‘the specified project 
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APPENDIX B 


LIST OF PROJECTSS STUDIED 


This Appendix lists the projects studied by identi- 
Picatlonenumber andmaSnoreetracie, mand includes a paragraph 
describing each. This information was available to each 


judge when he compiled his ranking. 


P.O), Ce to NO. eo Olaime enh C 
1 Drug Abuse 


This area consists of research and develop- 
Ment tomm@mceteaminime @auses, attitudes toward, and 
extent of drug abuse in the Navy; (b) exploration of means 
to reduce, prevent, and control drue abuse; and (c) means 
Pomp rermac eitectave education, counseling, and rehabili- 
tation programs for Naval personnel and dependents. 


2, Education and Training 


The two mutually supporting purposes of 
Pome ewoneandsirainine xsesearch and development are (a) 
improving Navy training and (b) development of principles 
and technology of training. Navy requirements call for 
more accurate definition of training requirements and 
for training methods directed toward meeting future job 
requirements. Research and development in this area in- 
volves work on the broad spectrum of problems related 
deriving and specifying appropriate training content and 
developing and evaluating ways to improve acquisition 
of training objectives. Research in this area also 
includes specific problems of learning, perception, 
job aids, programming, displays, simulation, and manage- 
ment as they relate to the improvement of training. 


3 Occupational Research and Development 


Development, test and evaluation of new 
and improved techniques for describing, evaluating, 
Structuring, and re-engineering Navy occupations for 
Speieeeeiatening of available manpower characteristics 
to changing job requirements. Involves new methods for 
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determining skills required for effective performance of 
eccupattons and tle*®storage"and retrieval of occupational 
data pertaining to work requirements of Navy personnel. 


4 Manpower/Personnel Cost Systems 


Development, tiest "andlevaluation of new 
and improved (a) techniques for the acquisition, computa- 
tion, storage, and retrieval of manpower/personnel cost 
data; (b) techniques for determining the cost of various 
mixes of men and equipment, personnel management policies, 
and personnel utilization factors; and (c) measure’s which 
permit manpower requirements planners to trade off cost 
and effectiveness in order to evaluate available options. 
The? principal objectivetws te furnismeeeoca leoovernment 
costs from procurement through post-sepearation which can 
be used to formulate cost-related decisions concerning the 
total Navy manpower personnel systems. 


5 Manpower Requirements 


Development, test and evaluation of 
Cechniique’s’ for defainingvand displlayine gquantitativerwand 
qualitative manpower requirements for all functions per- 
formed ashore and afloat including upward aggregation of 
requirements to the total Navy establishment. The tech- 
niques must be responsive to the changing operational 
requirements, system design innovations, and available 
human resources. Includes the development of manpower/ 
productivity models and the use of simulation techniques 
to test alternative manning and manpower management con- 
cepts in terms of cost and total syStem effectiveness. 
The principal obje@ectivewisetoe develop methods for deter- 
mining manpower requirements which are capable of respond- 
ing quickly to changes in resources and military objectives 
and requirements. 


6 Personnel Management Systems 


Development, test and evaluation of tools 
to improve the quality of personnel management by increas- 
ing the accuracy of force projections, by responding more 
rapidly to changing policies and operational conditions, 
and by projecting the effects of current and proposed per- 
sonnel policies to determine the most effective personnel 
Management strategies. These tools are usually in the form 
of computerized models of the personnel system which can be 
used to reflect the relationship between such variables as 
recruleeriVvaencenentem aid ateEritcton and to project the 
effect of these variables upon the size and skill composi- 
tion of the personnel force. 
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7 Man-Machine R & D 


This yarea, ySeprimariily concerned with the 
implementation of human operator considerations in the 
development, operations, and maintenance of new and cur- 
rent organizational, weapons and Support systems. Goals 
include (a) an understanding of, and techniques to quantify 
and extrapolate, meaningful relationships among human per- 
formance, equipment characteristics, and environmental 
conditions; (b) development and application of man-machine 
Standards, specifications, and tradeoff analysis methodolo- 
gies in systems development, planning, and procurement; and 
(c) techniques for application of human capabiliitiés to 
anticipated demand of future weapon systems. 


8 Sia laced ony.6 laisishi ta gation 


The program area involves development, 
test and evaluation of new and improved methods for (a) 
more accurate and effective selection and classification 
of individuals; (b) determining their training requirements; 
and (c) matching the man to the job. Selection usually 
refers to the process of determining which individual or 
individuals should be accepted or rejected with regard to 
a specific program. Classification refers to determining 
and categorizing the appropriate skill qualifications of 
Gimmivaivadual that each school or jib quota is filled 
with particular individuals most likely to perform success- 
fraolly. 


g Applied Systems Development 


Piemimhtocatlon of personnel and training 
requirements, establishment of initial operator performance 
Standards and maintenance skill requirements, assistance 
in man-machine tradeoff studies, and conducting studies to 
enhance man-machine compatibility for candidate and proto- 
type systems under development. The effort frequently 
involves development of training objectives, review of 
course and lesson plans, and the evaluation of operator 
and maintenance personnel training. 


10 Gareer Structure 


Development, test and evaluation of new 
and improved techniques for (a) arriving at optimal career 
progression patterns; (b) methods for utilizing occupational, 
Manpower/personnel cost, and other manpower factors data 
in determining optimal combination of grade and skill 
profiles required to satisfy Navy operational requirements; 
and (c) techniques for providing balance between needs of 
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the service based on technological projections, and needs 
of the individual related to opportunities for increased 
responsibility, advancement, prestige, and compensation. 
Typical outputs are optimum length of careers, optimal 
level of specialization, and optimum amount of fleet ex-. 
perience required of each grade of occupational specialty 
to meet changing military requirements. 


aie Personnel Retention 


This is a multi-faceted program exploring 
and developing methods and techniques for inducing quali- 
fied and needed personnel to remain on active duty, and 
programs which meet fluctuating needs of the naval service. 
Included are (a) investigation in psychometric technology 
to predict retention for use in selection and assignment; 
(b) development of more fair and effective evaluation of 
performance through computer-aided analysis and display of 
performance evaluation; and (c) development of improved 
techniques for selection and training of personnel in 
ratings which administer personal services to other naval 
personnel. 


dk? Attitude/Motivation 


Daa opm csmeciometric and psychometric in- 
Strumemtseand conduct controlled experiments for purposes 
of studying man's values, needs, aspirations, prejudices 
and expectations to achieve a better understanding of how 
te wenie chehis thinking, feelings, and reactions to 
DeomlLewrcLOUpSs, SOCGCial istsues or events in his environment. 
Pie moofecChive Gteachievyine such an understanding is better 
fieeommetlonmand control of an individual's behavior, moti- 
vation and performance. 


1} Minority Discrimination Measurement and 
Reduction 


This area is comprised of R & D support 
for BuPers and Navy objectives to intensify and expand 
Pere smoucliminate racial, sex, and cultural discrimina- 
tion and bias in the Navy. 


14 Organization Effectiveness 


Development and evaluation of techniques 
for measuring overall military effectiveness of organiza- 
Pe units under various contingencies. This involves 
the development of empirically based criteria and the 
memmeive importance of criteria which should be applied 
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in making judgments of organizational effectiveness. It 
includes studies of individual and organizational behavior 
and evaluative research into managerial practices and or- 
ganizational characteristics. The principal objective is 
the enhancement of organizational effectiveness in rela- 
tion to organizational goals and total system effectiveness. 


15 Distribution/Assignment 


This area encompasses research and 
development in the managerial functions of personnel 
disstribution, assignment, and rotation as carried out by 
the Bureau of Naval Personnel. The program is concerned 
with the development and evaluation of new and improved 
techniques and procedures based on modern computer 
technology for (a) the equitable, quantitative and 
Gualitative ‘allocation ("distribution") of the pérsonnel 
and inventory among the ships, aircraft squadrons, and 
shore stations of the Navy; (b) the optimal match 
("assignment") of specific individuals to particular 
bilVets in terms of prescribed criteria; and (c) the 
Planned, periodic reassignment ("rotation") of career 
personnel to the various categories of sea, shore and 
overseas duty. Objectives are the optimum matching 
of personnel and billets, improved manpower utilization 
Systems, and optimal matching of manpower requirements 
and resources over time. 


16 R & D Resources Management 


This area includes three subareas dealing 


femme fficient use of R & D resources. One deals with 
Mmemeselection and evaluation of scientific/research per- 
sonnel employed by the laboratories. Research in this 


area is aimed at the development of performance evaluation 
criteria in the area of quantitative and qualitative 

merk output and scientific/technical contributions. 
Another subarea concerns the identification of significant 
problem areas, assignment of priorities, and selection of 
tasks to be performed to assure maximum return for the"R & 
M@edolilar”. A third subarea is the continuing study of the 
implementation of research recommendations by consumers as 
a basis for future research planning and to facilitate the 
implementation of significant R & D products. 


17 Recruitment/Procurement 


Development, test and evaluation of 
methods and techniques for (a) attracting and obtaining 
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mequared sSxill composi Ci 6n.wtac ial Mix,eand i@ducazional 
bevels and for (b) improving selection policies and 
programs relating to manpower requirements, implications 
of an all volunteer force, manpower ceilings reductions, 
and changes in strategic guidance. The principal 
objective is to obtain personnel with the appropriate 
innate and acquired characteristics to meet alli of the 
performance requirements and changing requirements of 
future naval forces. 


18 Personnel Performance 


Development of on-the-job performance 
standards and criteria and improved proficiency measure- 
ment techniques to assure that selection, assignment, 
and promotion factors are job-relevant. The goal is 
Maximum readiness of individuals and units. 
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MATRIX M 
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APPENDIX D 
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